1. Technical Field
The present invention relates to artificial neural networks, and more particularly to proofreading connection adjustments in hardware artificial neural networks.
2. Discussion of Related Art
Artificial Neural Networks (ANNs) are powerful computational tools loosely based on information processing in the brain, with widespread applications. The key ingredients in ANNs are (1) neuron-like signaling elements (“neuroids”) interconnected by variable-strength connections (analogous to synapses) and giving outputs that represent computational results, (2) the neuroids typically compute weighted sums of their inputs (although their outputs may be some nonlinear function of these weighted sums), and (3) connection strengths can be modified as a result of ongoing computations so that the ANN “learns” to perform the desired computation. The core operations performed by a neuroid are the weighted input sum operation and the weight update operation. The weighted input sum operation is a “dot product” representing an Euclidian measure of the similarity between the current input vector and the stored weight vector, while the weight update operation allows improvements in performance by learning.
Although most current ANNs are implemented in software in conventional serial computers, ANNs can be implemented using a variety of techniques, such as in massively parallel hardware so that the resulting hANN (hardware ANN) compute vast numbers of high dimensional dot products and weight improvements essentially instantaneously, like the brain.
Implementation in hardware inevitably means that some of the steps in the computations may be imperfectly executed, a possibility which is not permitted in conventional Turing-von Neumann serial computers. Indeed, the inevitably of noise and errors in nanoscale components is one reason why neuromorphic, hANN approaches have been advocated, since they are more noise and error-resistant than conventional approaches.
Regardless of the implementation, a key step in most ANNs is training by adjusting connection strengths according to some well-defined recipe. One known recipe is the generalized Delta rule (part of “back propagation”), in which a connection changes according to the product of its input value and the difference between its current and desired output value. The closely related “general Hebbian” rule is also widespread, where changes are made according to the product of the input value and some (possibly nonlinear function of) the output value.
In all cases, if ANNs are to be realized in hardware, the connections, and their learning rule, must be physically implemented. This means that the learning rule, whether Delta or Hebb or something else, must be local in that the signals that produce the connection adjustment must be present at the actual connections themselves. Of course, this is not difficult to achieve, since by definition the connection brings together the input and output signals, the key components of the update rule. The term “Hebbian” is used generally herein to cover all cases where the local update depends on the local activity.
In any hardware implementation any massively parallel ANN will have a high density of physical connections, where physically neighboring connections might influence each other, so that the learning rule will not be completely local. For example, heat generated at the connections either during the readout process, or, more likely, during updates (typically the energy needed to update is much higher than the energy needed to read out, ensuring that readout alone does not perturb connection strengths), could travel along connecting wires to “nearby” connections. Since massively parallel hANNs must be efficiently wired, wires will be short and therefore good heat conductors. The slight nonlocality of the learning rule in hANNs has no particular relation to the task the hANN is designed to perform: it will inevitably degrade performance, and might completely prevent useful operation. This problem has hitherto been somewhat neglected in practical ANNs, since in serial implementations it does not arise, while in prototype hANN devices the connection density has been low as to not incur non-local influence. A second, related, problem is that in physically small (e.g., nanoscale) connections, strength adjustments may be a noisy function of the current input and output values even though the adjustments at individual connections may occur independently. While careful design will minimize this “update noise” problem, eliminating it will increase the bulk of the individual connections, lowering the power of the massively parallel ANN.
Therefore, a need exists for proofreading connection adjustments in hardware artificial neural networks.